Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet Indexer Data Scripts #157

Merged
merged 7 commits into from
Dec 20, 2024
Merged

Parquet Indexer Data Scripts #157

merged 7 commits into from
Dec 20, 2024

Conversation

marcus-snx
Copy link
Collaborator

@marcus-snx marcus-snx commented Dec 17, 2024 •
edited
Loading

To get Core models in ClickHouse:

  1. start clickhouse:
    docker compose up clickhouse -d
  2. start indexer:
    docker compose run -e NETWORK_NAME={network_name} -e PROTOCOL_NAME={protocol_name} -e BLOCK_FROM={block_from} indexer
  3. clean raw data:
    docker compose run -e NETWORK_NAME={network_name} -e PROTOCOL_NAME={protocol_name} indexer uv run scripts/clean_parquet.py
  4. import clean data to ClickHouse:
    docker compose run -e NETWORK_NAME={network_name} -e PROTOCOL_NAME={protocol_name} indexer uv run scripts/import_parquet.py
  5. run dbt models:
    docker compose run transformer dbt run --target prod --profiles-dir profiles --profile clickhouse --select tag:{network_name}
  6. run dbt tests:
    docker compose run transformer dbt test --target prod --profiles-dir profiles --profile clickhouse --select tag:{network_name}

@marcus-snx marcus-snx added the wip Work in progress label Dec 17, 2024
@marcus-snx marcus-snx requested a review from Tburm December 19, 2024 00:07
@marcus-snx marcus-snx removed the wip Work in progress label Dec 19, 2024
@Tburm
Copy link
Collaborator

Tburm commented Dec 19, 2024

I did update a bunch of things based on my own PR, but I really can't get the import to run properly on that branch. I'm seeing this error, even though the tables do exist in my clickhouse instance when I inspect it manually:

Error inserting data into raw_base_mainnet.synthetix_perps_market_proxy_event_order_committed: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 60
 Code: 60. DB::Exception: Table raw_base_mainnet.synthetix_perps_market_proxy_event_order_committed does not exist. Maybe you meant raw_base_mainnet.synthetix_spot_market_proxy_event_order_committed?. (UNKNOWN_TABLE) (version 24.11.1.2557 (official build))

Error inserting data into raw_base_mainnet.synthetix_spot_market_proxy_event_atomic_fixed_fee_set: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 60
 Code: 60. DB::Exception: Table raw_base_mainnet.synthetix_spot_market_proxy_event_atomic_fixed_fee_set does not exist. Maybe you meant raw_base_mainnet.synthetix_spot_market_proxy_event_async_fixed_fee_set?. (UNKNOWN_TABLE) (version 24.11.1.2557 (official build))

Error inserting data into raw_base_mainnet.synthetix_core_proxy_event_upgraded: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 60
 Code: 60. DB::Exception: Table raw_base_mainnet.synthetix_core_proxy_event_upgraded does not exist. Maybe you meant raw_base_mainnet.synthetix_account_proxy_event_upgraded?. (UNKNOWN_TABLE) (version 24.11.1.2557 (official build))

Error inserting data into raw_base_mainnet.synthetix_perps_account_proxy_event_approval: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 60
 Code: 60. DB::Exception: Table raw_base_mainnet.synthetix_perps_account_proxy_event_approval does not exist. Maybe you meant raw_base_mainnet.synthetix_account_proxy_event_approval?. (UNKNOWN_TABLE) (version 24.11.1.2557 (official build))

Error inserting data into raw_base_mainnet.synthetix_account_proxy_event_upgraded: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 60
 Code: 60. DB::Exception: Table raw_base_mainnet.synthetix_account_proxy_event_upgraded does not exist. Maybe you meant raw_base_mainnet.synthetix_account_proxy_event_approval?. (UNKNOWN_TABLE) (version 24.11.1.2557 (official build))

Error inserting data into raw_base_mainnet.synthetix_perps_market_proxy_event_order_settled: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 60
 Code: 60. DB::Exception: Table raw_base_mainnet.synthetix_perps_market_proxy_event_order_settled does not exist. Maybe you meant raw_base_mainnet.synthetix_spot_market_proxy_event_order_settled?. (UNKNOWN_TABLE) (version 24.11.1.2557 (official build))

Any idea why this is happening @marcus-snx ?

@Tburm Tburm merged commit 52aa601 into feat/clickhouse-raw-models Dec 20, 2024
1 check passed
@Tburm Tburm deleted the feat/indexer-scripts branch December 20, 2024 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Tburm Tburm Awaiting requested review from Tburm

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@marcus-snx @Tburm